Empirical entropy in context

In statistics as in life, many things become clearer when we consider context. Statisticians’ use of context itself becomes clearer, in fact, when we consider the past century. It was anathema to them prior to 1906, when Markov [22] proved the weak law of large numbers applies to chains of dependent events over finite domains (i.e., finite-state Markov processes). He published several papers on the statistics of dependent events and in 1913 gave an example of dependence in language: he analyzed the first 20 000 characters of Pushkin’s Eugene Onegin and found the likelihood of a vowel was strongly affected by the presence of vowels in the four preceding positions. Many other examples have been found since, in physics, chemistry, biology, economics, sociology, psychology — every branch of the natural and social sciences. While Markov was developing the idea of Markov processes, another probability theorist, Borel, was starting an investigation into examples beyond their scope. Borel [2] defined a number to be normal in base b if, in its infinite b-ary representation, every k-tuple occurs with relative frequency 1/b; he called a number absolutely normal if normal in every base. Using the Borel-Cantelli Lemma, he showed nearly all numbers are absolutely normal, although his proof was completely non-constructive. Sierpinski [28] gave the first example of an absolutely normal number but his construction is still not known to be computable. The first concrete example of a normal number (although not absolutely normal) was found by Champernowne [5], who proved 0 . 1 2 3 4 5 6 7 8 9 10 11 12 . . . — the concatenation of the integers — is normal in decimal. Champernowne published his number while still an undergraduate at Cambridge, where he would certainly have known of Markov processes from studying economics with Keynes. Champernowne was a close friend of Turing, who would also have known of Markov processes from studying statistics with Hardy, and Turing [31] tried to find a concrete example of an absolutely normal number. It seems quite possible the definition of a Markov process influenced Turing’s definition of a universal machine (i.e., Turing machine), and his search for an absolutely normal number influenced his study of incomputable numbers. After all, Markov processes were good models for the behaviourist psychology popular in the 1930s, Turing is famous for studying computational models of the human mind, and the existence of computable normal numbers shows Turing machines’ tapes increase their computational power.